With the advance of digital technologies, especially the widespread use and availability of digital camcorders, digital video is getting easier and more efficient to use in a wide variety of applications, such as entertainment, education, medicine, security, and military. Accordingly, there is an increasing demand for video processing techniques, such as noise reduction.
There is always certain level of noise captured in a video sequence. The sources are numerous, including electronic noise, photon noise, film grain noise, and quantization noise. The noise adversely affects video representation, storage, display, and transmission. It contaminates visual quality, decreases coding efficiency (with increased entropy), increases transmission bandwidth, and makes content description less discriminative and effective. Therefore, it is desirable to reduce the noise while preserving video content.
After years of effort, video filtering still remains as a challenging task. Most of the time, the only information available is the input noisy video. Neither the noise-free video nor the error characteristics are available. To effectively reduce the random noise, motion estimation is necessary to enhance temporal correlation, by establishing point correspondence between video frames. However, motion estimation itself is an under-constrained and ill-posed problem, especially when there is noise involved. Perfect motion estimation is almost impossible or not practical. Meanwhile, spatiotemporal filtering is also necessary to actually reduce the random noise. The filter design heavily depends on the knowledge of the noise characteristics (which are usually not available). Furthermore, video processing requires tremendous computational power because of the amount of data involved.
Research on noise estimation and reduction in a video sequence has been going on for decades. “Noise reduction in image sequence using motion-compensated temporal filtering” by E. Dubois and M. Sabri, IEEE Trans. on Communication, 32(7):826-831, 1984, presented one of the earliest schemes using motion for noise reduction. A comprehensive review of various methods is available in “Noise reduction filters for dynamic image sequence: a review” by J. C. Brailean, et al., Proceedings of the IEEE, 83(9):1272—1292, September 1995. A robust motion estimation algorithm is presented in “The robust estimation of multiple motions: parametric and piecewise smooth flow fields” by M. Black and P. Anandan, Computer Vision and Image Understanding, 63:75-104, January 1996.
In addition, the following patent publications bear some relevance to this area; each of which are incorporated herein by reference. Commonly-assigned U.S. Published Patent Application No. 20020109788, “Method and system for motion image digital processing” by R. Morton et al., discloses a method to reduce film grain noise in digital motion signals by using a frame averaging technique. A configuration of successive motion estimation and noise removal is employed. U.S. Pat. No. 6,535,254, “Method and device for noise reduction” to K. Olsson et al., discloses a method of reducing noise in a video signal. U.S. Pat. No. 6,281,942, “Spatial and temporal filtering mechanism for digital motion video signals” to A. Wang, discloses a digital motion video processing mechanism of adaptive spatial filtering followed by temporal filtering of video frames. U.S. Pat. No. 5,909,515, “Method for the temporal filtering of the noise in an image of a sequence of digital images, and device for carrying out the method” to S. Makram-Ebeid, discloses a method for temporal filtering of a digital image sequence. Separate motion and filtering steps were taken in a batch mode to reduce noise. U.S. Pat. No. 5,764,307, “Method and apparatus for spatially adaptive filtering for video encoding” to T. Ozcelik et al., discloses a method and an apparatus for spatially adaptive filtering a displaced frame difference and reducing the amount of information that must be encoded by a video encoder without substantially degrading the decoded video sequence. The filtering is carried out in the spatial domain on the displaced frames (the motion compensated frames). The goal is to facilitate video coding, so that the compressed video has reduced noise (and smoothed video content as well). U.S. Pat. No. 5,600,731, “Method for temporally adaptive filtering of frames of a noisy image sequence using motion estimation” to M. I. Sezan et al., discloses a temporally adaptive filtering method to reduce noise in an image sequence. Commonly-assigned U.S. Pat. No. 5,384,865, “Adaptive, hybrid median filter for temporal noise suppression” to J. Loveridge, discloses a temporal noise suppression scheme utilizing median filtering upon a time-varying sequence of images.
In addition, International Publication No. WO94/09592, “Three dimensional median and recursive filtering for video image enhancement” to S. Takemoto et al., discloses methods for video image enhancement by spatiotemporal filtering with or without motion estimation. International Publication No. WO01/97509, “Noise filtering an image sequence” to W. Bruls et al., discloses a method to filter an image sequence with the use of estimated noise characteristics. Published European Patent Application EP0840514, “Method and apparatus for prefiltering of video images” to M. Van Ackere et al., discloses a method for generating an updated video stream with reduced noise for video encoding applications. European Patent Specification EP0614312, “Noise reduction system using multi-frame motion estimation, outlier rejection and trajectory correction” to S.-L. Iu, discloses a noise reduction system.
One of the common features of the previously disclosed schemes is the use of independent and separate steps of motion estimation and spatiotemporal filtering. Motion estimation is taken as a preprocessing step in a separate module before filtering, and there is no interaction between the two modules. If the motion estimation fails, filtering is carried out on a collection of uncorrelated samples, and there is no way to recover from such a failure. Also there is no attempt to explicitly estimate the noise levels, leading to a high chance of mismatch between the noise in the video and the algorithms and the parameters used for noise reduction. Furthermore, a robust method has not been used in video filtering, and the performance suffers when the underlying model and assumptions are violated occasionally, which happens when that data is corrupted by noise.